Systemd - Journalctl
How To Use Systemctl to Manage Systemd Services and Units
Starting and Stopping Services
Enabling and Disabling Services
Checking the Status of Services
Adjusting the System State (Runlevel) with Targets
Getting and Setting the Default Target
Using Shortcuts for Important Events
How To Use Journalctl to View and Manipulate Systemd Logs
Displaying Logs from the Current Boot
Systemd : Utiliser journalctl, les logs de systemd
Visualiser les logs : Utiliser journalctl
Envoyer quand même dans /var/log/messages
Systemd is an init system and system manager that is widely becoming the new standard for Linux machines. While there are considerable opinions about whether systemd is an improvement over the traditional SysV init systems it is replacing, the majority of distributions plan to adopt it or have already done so.
Due to its heavy adoption, familiarizing yourself with systemd is well worth the trouble, as it will make administering servers considerably easier. Learning about and utilizing the tools and daemons that comprise systemd will help you better appreciate the power, flexibility, and capabilities it provides, or at least help you to do your job with minimal hassle.
In this guide, we will be discussing the systemctl command, which is the central management tool for controlling the init system. We will cover how to manage services, check statuses, change system states, and work with the configuration files.
Please note that although systemd has become the default init system for many Linux distributions, it isn’t implemented universally across all distros. As you go through this tutorial, if your terminal outputs the error bash: systemctl is not installed then it is likely that your machine has a different init system installed.
The fundamental purpose of an init system is to initialize the components that must be started after the Linux kernel is booted (traditionally known as “userland” components). The init system is also used to manage services and daemons for the server at any point while the system is running. With that in mind, we will start with some simple service management operations.
In systemd, the target of most actions are “units”, which are resources that systemd knows how to manage. Units are categorized by the type of resource they represent and they are defined with files known as unit files. The type of each unit can be inferred from the suffix on the end of the file.
For service management tasks, the target unit will be service units, which have unit files with a suffix of .service. However, for most service management commands, you can actually leave off the .service suffix, as systemd is smart enough to know that you probably want to operate on a service when using service management commands.
To start a systemd service, executing instructions in the service’s unit file, use the start command. If you are running as a non-root user, you will have to use sudo since this will affect the state of the operating system:
sudo systemctl start application.service
As we mentioned above, systemd knows to look for *.service files for service management commands, so the command could just as easily be typed like this:
sudo systemctl start application
Although you may use the above format for general administration, for clarity, we will use the .service suffix for the remainder of the commands to be explicit about the target we are operating on.
To stop a currently running service, you can use the stop command instead:
sudo systemctl stop application.service
To restart a running service, you can use the restart command:
sudo systemctl restart application.service
If the application in question is able to reload its configuration files (without restarting), you can issue the reload command to initiate that process:
sudo systemctl reload application.service
If you are unsure whether the service has the functionality to reload its configuration, you can issue the reload-or-restart command. This will reload the configuration in-place if available. Otherwise, it will restart the service so the new configuration is picked up:
sudo systemctl reload-or-restart application.service
The above commands are useful for starting or stopping commands during the current session. To tell systemd to start services automatically at boot, you must enable them.
To start a service at boot, use the enable command:
sudo systemctl enable application.service
This will create a symbolic link from the system’s copy of the service file (usually in /lib/systemd/system or /etc/systemd/system) into the location on disk where systemd looks for autostart files (usually /etc/systemd/system/some_target.target.wants. We will go over what a target is later in this guide).
To disable the service from starting automatically, you can type:
sudo systemctl disable application.service
This will remove the symbolic link that indicated that the service should be started automatically.
Keep in mind that enabling a service does not start it in the current session. If you wish to start the service and enable it at boot, you will have to issue both the start and enable commands.
To check the status of a service on your system, you can use the status command:
systemctl status application.service
This will provide you with the service state, the cgroup hierarchy, and the first few log lines.
For instance, when checking the status of an Nginx server, you may see output like this:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2015-01-27 19:41:23 EST; 22h ago
Main PID: 495 (nginx)
CGroup: /system.slice/nginx.service
├─495 nginx: master process /usr/bin/nginx -g pid /run/nginx.pid; error_log stderr;
└─496 nginx: worker process
Jan 27 19:41:23 desktop systemd[1]: Starting A high performance web server and a reverse proxy server...
Jan 27 19:41:23 desktop systemd[1]: Started A high performance web server and a reverse proxy server.
This gives you a nice overview of the current status of the application, notifying you of any problems and any actions that may be required.
There are also methods for checking for specific states. For instance, to check to see if a unit is currently active (running), you can use the is-active command:
systemctl is-active application.service
This will return the current unit state, which is usually active or inactive. The exit code will be “0” if it is active, making the result simpler to parse programmatically.
To see if the unit is enabled, you can use the is-enabled command:
systemctl is-enabled application.service
This will output whether the service is enabled or disabled and will again set the exit code to “0” or “1” depending on the answer to the command question.
A third check is whether the unit is in a failed state. This indicates that there was a problem starting the unit in question:
systemctl is-failed application.service
This will return active if it is running properly or failed if an error occurred. If the unit was intentionally stopped, it may return unknown or inactive. An exit status of “0” indicates that a failure occurred and an exit status of “1” indicates any other status.
The commands so far have been useful for managing single services, but they are not very helpful for exploring the current state of the system. There are a number of systemctl commands that provide this information.
To see a list of all of the active units that systemd knows about, we can use the list-units command:
systemctl list-units
This will show you a list of all of the units that systemd currently has active on the system. The output will look something like this:
UNIT LOAD ACTIVE SUB DESCRIPTION
atd.service loaded active running ATD daemon
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
dbus.service loaded active running D-Bus System Message Bus
dcron.service loaded active running Periodic Command Scheduler
dkms.service loaded active exited Dynamic Kernel Modules System
getty@tty1.service loaded active running Getty on tty1
. . .
The output has the following columns:
•.UNIT: The systemd unit name
•.LOAD: Whether the unit’s configuration has been parsed by systemd. The configuration of loaded units is kept in memory.
•.ACTIVE: A summary state about whether the unit is active. This is usually a fairly basic way to tell if the unit has started successfully or not.
•.SUB: This is a lower-level state that indicates more detailed information about the unit. This often varies by unit type, state, and the actual method in which the unit runs.
•.DESCRIPTION: A short textual description of what the unit is/does.
Since the list-units command shows only active units by default, all of the entries above will show “loaded” in the LOAD column and “active” in the ACTIVE column. This display is actually the default behavior of systemctl when called without additional commands, so you will see the same thing if you call systemctl with no arguments:
systemctl
We can tell systemctl to output different information by adding additional flags. For instance, to see all of the units that systemd has loaded (or attempted to load), regardless of whether they are currently active, you can use the --all flag, like this:
systemctl list-units --all
This will show any unit that systemd loaded or attempted to load, regardless of its current state on the system. Some units become inactive after running, and some units that systemd attempted to load may have not been found on disk.
You can use other flags to filter these results. For example, we can use the --state= flag to indicate the LOAD, ACTIVE, or SUB states that we wish to see. You will have to keep the --all flag so that systemctl allows non-active units to be displayed:
systemctl list-units --all --state=inactive
Another common filter is the --type= filter. We can tell systemctl to only display units of the type we are interested in. For example, to see only active service units, we can use:
systemctl list-units --type=service
The list-units command only displays units that systemd has attempted to parse and load into memory. Since systemd will only read units that it thinks it needs, this will not necessarily include all of the available units on the system. To see every available unit file within the systemd paths, including those that systemd has not attempted to load, you can use the list-unit-files command instead:
systemctl list-unit-files
Units are representations of resources that systemd knows about. Since systemd has not necessarily read all of the unit definitions in this view, it only presents information about the files themselves. The output has two columns: the unit file and the state.
UNIT FILE STATE
proc-sys-fs-binfmt_misc.automount static
dev-hugepages.mount static
dev-mqueue.mount static
proc-fs-nfsd.mount static
proc-sys-fs-binfmt_misc.mount static
sys-fs-fuse-connections.mount static
sys-kernel-config.mount static
sys-kernel-debug.mount static
tmp.mount static
var-lib-nfs-rpc_pipefs.mount static
org.cups.cupsd.path enabled
. . .
The state will usually be “enabled”, “disabled”, “static”, or “masked”. In this context, static means that the unit file does not contain an “install” section, which is used to enable a unit. As such, these units cannot be enabled. Usually, this means that the unit performs a one-off action or is used only as a dependency of another unit and should not be run by itself.
We will cover what “masked” means momentarily.
So far, we have been working with services and displaying information about the unit and unit files that systemd knows about. However, we can find out more specific information about units using some additional commands.
To display the unit file that systemd has loaded into its system, you can use the cat command (this was added in systemd version 209). For instance, to see the unit file of the atd scheduling daemon, we could type:
systemctl cat atd.service
[Unit]
Description=ATD daemon
[Service]
Type=forking
ExecStart=/usr/bin/atd
[Install]
WantedBy=multi-user.target
The output is the unit file as known to the currently running systemd process. This can be important if you have modified unit files recently or if you are overriding certain options in a unit file fragment (we will cover this later).
To see a unit’s dependency tree, you can use the list-dependencies command:
systemctl list-dependencies sshd.service
This will display a hierarchy mapping the dependencies that must be dealt with in order to start the unit in question. Dependencies, in this context, include those units that are either required by or wanted by the units above it.
sshd.service
├─system.slice
└─basic.target
├─microcode.service
├─rhel-autorelabel-mark.service
├─rhel-autorelabel.service
├─rhel-configure.service
├─rhel-dmesg.service
├─rhel-loadmodules.service
├─paths.target
├─slices.target
. . .
The recursive dependencies are only displayed for .target units, which indicate system states. To recursively list all dependencies, include the --all flag.
To show reverse dependencies (units that depend on the specified unit), you can add the --reverse flag to the command. Other flags that are useful are the --before and --after flags, which can be used to show units that depend on the specified unit starting before and after themselves, respectively.
To see the low-level properties of a unit, you can use the show command. This will display a list of properties that are set for the specified unit using a key=value format:
systemctl show sshd.service
Id=sshd.service
Names=sshd.service
Requires=basic.target
Wants=system.slice
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=syslog.target network.target auditd.service systemd-journald.socket basic.target system.slice
Description=OpenSSH server daemon
. . .
If you want to display a single property, you can pass the -p flag with the property name. For instance, to see the conflicts that the sshd.service unit has, you can type:
systemctl show sshd.service -p Conflicts
Conflicts=shutdown.target
We saw in the service management section how to stop or disable a service, but systemd also has the ability to mark a unit as completely unstartable, automatically or manually, by linking it to /dev/null. This is called masking the unit, and is possible with the mask command:
sudo systemctl mask nginx.service
This will prevent the Nginx service from being started, automatically or manually, for as long as it is masked.
If you check the list-unit-files, you will see the service is now listed as masked:
systemctl list-unit-files
. . .
kmod-static-nodes.service static
ldconfig.service static
mandb.service static
messagebus.service static
nginx.service masked
quotaon.service static
rc-local.service static
rdisc.service disabled
rescue.service static
. . .
If you attempt to start the service, you will see a message like this:
sudo systemctl start nginx.service
Failed to start nginx.service: Unit nginx.service is masked.
To unmask a unit, making it available for use again, simply use the unmask command:
sudo systemctl unmask nginx.service
This will return the unit to its previous state, allowing it to be started or enabled.
While the specific format for unit files is outside of the scope of this tutorial, systemctl provides built-in mechanisms for editing and modifying unit files if you need to make adjustments. This functionality was added in systemd version 218.
The edit command, by default, will open a unit file snippet for the unit in question:
sudo systemctl edit nginx.service
This will be a blank file that can be used to override or add directives to the unit definition. A directory will be created within the /etc/systemd/system directory which contains the name of the unit with .d appended. For instance, for the nginx.service, a directory called nginx.service.d will be created.
Within this directory, a snippet will be created called override.conf. When the unit is loaded, systemd will, in memory, merge the override snippet with the full unit file. The snippet’s directives will take precedence over those found in the original unit file.
If you wish to edit the full unit file instead of creating a snippet, you can pass the --full flag:
sudo systemctl edit --full nginx.service
This will load the current unit file into the editor, where it can be modified. When the editor exits, the changed file will be written to /etc/systemd/system, which will take precedence over the system’s unit definition (usually found somewhere in /lib/systemd/system).
To remove any additions you have made, either delete the unit’s .d configuration directory or the modified service file from /etc/systemd/system. For instance, to remove a snippet, we could type:
sudo rm -r /etc/systemd/system/nginx.service.d
To remove a full modified unit file, we would type:
sudo rm /etc/systemd/system/nginx.service
After deleting the file or directory, you should reload the systemd process so that it no longer attempts to reference these files and reverts back to using the system copies. You can do this by typing:
sudo systemctl daemon-reload
Targets are special unit files that describe a system state or synchronization point. Like other units, the files that define targets can be identified by their suffix, which in this case is .target. Targets do not do much themselves, but are instead used to group other units together.
This can be used in order to bring the system to certain states, much like other init systems use runlevels. They are used as a reference for when certain functions are available, allowing you to specify the desired state instead of the individual units needed to produce that state.
For instance, there is a swap.target that is used to indicate that swap is ready for use. Units that are part of this process can sync with this target by indicating in their configuration that they are WantedBy= or RequiredBy= the swap.target. Units that require swap to be available can specify this condition using the Wants=, Requires=, and After= specifications to indicate the nature of their relationship.
The systemd process has a default target that it uses when booting the system. Satisfying the cascade of dependencies from that single target will bring the system into the desired state. To find the default target for your system, type:
systemctl get-default
multi-user.target
If you wish to set a different default target, you can use the set-default. For instance, if you have a graphical desktop installed and you wish for the system to boot into that by default, you can change your default target accordingly:
sudo systemctl set-default graphical.target
You can get a list of the available targets on your system by typing:
systemctl list-unit-files --type=target
Unlike runlevels, multiple targets can be active at one time. An active target indicates that systemd has attempted to start all of the units tied to the target and has not tried to tear them down again. To see all of the active targets, type:
systemctl list-units --type=target
It is possible to start all of the units associated with a target and stop all units that are not part of the dependency tree. The command that we need to do this is called, appropriately, isolate. This is similar to changing the runlevel in other init systems.
For instance, if you are operating in a graphical environment with graphical.target active, you can shut down the graphical system and put the system into a multi-user command line state by isolating the multi-user.target. Since graphical.target depends on multi-user.target but not the other way around, all of the graphical units will be stopped.
You may wish to take a look at the dependencies of the target you are isolating before performing this procedure to ensure that you are not stopping vital services:
systemctl list-dependencies multi-user.target
When you are satisfied with the units that will be kept alive, you can isolate the target by typing:
sudo systemctl isolate multi-user.target
There are targets defined for important events like powering off or rebooting. However, systemctl also has some shortcuts that add a bit of additional functionality.
For instance, to put the system into rescue (single-user) mode, you can just use the rescue command instead of isolate rescue.target:
sudo systemctl rescue
This will provide the additional functionality of alerting all logged in users about the event.
To halt the system, you can use the halt command:
sudo systemctl halt
To initiate a full shutdown, you can use the poweroff command:
sudo systemctl poweroff
A restart can be started with the reboot command:
sudo systemctl reboot
These all alert logged in users that the event is occurring, something that simply running or isolating the target will not do. Note that most machines will link the shorter, more conventional commands for these operations so that they work properly with systemd.
For example, to reboot the system, you can usually type:
sudo reboot
By now, you should be familiar with some of the basic capabilities of the systemctl command that allow you to interact with and control your systemd instance. The systemctl utility will be your main point of interaction for service and system state management.
While systemctl operates mainly with the core systemd process, there are other components to the systemd ecosystem that are controlled by other utilities. Other capabilities, like log management and user sessions are handled by separate daemons and management utilities (journald/journalctl and logind/loginctl respectively). Taking time to become familiar with these other tools and daemons will make management an easier task.
Some of the most compelling advantages of systemd are those involved with process and system logging. When using other tools, logs are usually dispersed throughout the system, handled by different daemons and processes, and can be fairly difficult to interpret when they span multiple applications. Systemd attempts to address these issues by providing a centralized management solution for logging all kernel and userland processes. The system that collects and manages these logs is known as the journal.
The journal is implemented with the journald daemon, which handles all of the messages produced by the kernel, initrd, services, etc. In this guide, we will discuss how to use the journalctl utility, which can be used to access and manipulate the data held within the journal.
One of the impetuses behind the systemd journal is to centralize the management of logs regardless of where the messages are originating. Since much of the boot process and service management is handled by the systemd process, it makes sense to standardize the way that logs are collected and accessed. The journald daemon collects data from all available sources and stores them in a binary format for easy and dynamic manipulation.
This gives us a number of significant advantages. By interacting with the data using a single utility, administrators are able to dynamically display log data according to their needs. This can be as simple as viewing the boot data from three boots ago, or combining the log entries sequentially from two related services to debug a communication issue.
Storing the log data in a binary format also means that the data can be displayed in arbitrary output formats depending on what you need at the moment. For instance, for daily log management you may be used to viewing the logs in the standard syslog format, but if you decide to graph service interruptions later on, you can output each entry as a JSON object to make it consumable to your graphing service. Since the data is not written to disk in plain text, no conversion is needed when you need a different on-demand format.
The systemd journal can either be used with an existing syslog implementation, or it can replace the syslog functionality, depending on your needs. While the systemd journal will cover most administrator’s logging needs, it can also complement existing logging mechanisms. For instance, you may have a centralized syslog server that you use to compile data from multiple servers, but you also may wish to interleave the logs from multiple services on a single system with the systemd journal. You can do both of these by combining these technologies.
One of the benefits of using a binary journal for logging is the ability to view log records in UTC or local time at will. By default, systemd will display results in local time.
Because of this, before we get started with the journal, we will make sure the timezone is set up correctly. The systemd suite actually comes with a tool called timedatectl that can help with this.
First, see what timezones are available with the list-timezones option:
timedatectl list-timezones
This will list the timezones available on your system. When you find the one that matches the location of your server, you can set it by using the set-timezone option:
sudo timedatectl set-timezone zone
To ensure that your machine is using the correct time now, use the timedatectl command alone, or with the status option. The display will be the same:
timedatectl status
Local time: Thu 2015-02-05 14:08:06 EST
Universal time: Thu 2015-02-05 19:08:06 UTC
RTC time: Thu 2015-02-05 19:08:06
Time zone: America/New_York (EST, -0500)
NTP enabled: no
NTP synchronized: no
RTC in local TZ: no
DST active: n/a
The first line should display the correct time.
To see the logs that the journald daemon has collected, use the journalctl command.
When used alone, every journal entry that is in the system will be displayed within a pager (usually less) for you to browse. The oldest entries will be up top:
journalctl
-- Logs begin at Tue 2015-02-03 21:48:52 UTC, end at Tue 2015-02-03 22:29:38 UTC. --
Feb 03 21:48:52 localhost.localdomain systemd-journal[243]: Runtime journal is using 6.2M (max allowed 49.
Feb 03 21:48:52 localhost.localdomain systemd-journal[243]: Runtime journal is using 6.2M (max allowed 49.
Feb 03 21:48:52 localhost.localdomain systemd-journald[139]: Received SIGTERM from PID 1 (systemd).
Feb 03 21:48:52 localhost.localdomain kernel: audit: type=1404 audit(1423000132.274:2): enforcing=1 old_en
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules.
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules.
Feb 03 21:48:52 localhost.localdomain kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 8 users, 102 roles, 4976 types, 294 bools, 1 sens,
Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 83 classes, 104131 rules
. . .
You will likely have pages and pages of data to scroll through, which can be tens or hundreds of thousands of lines long if systemd has been on your system for a long while. This demonstrates how much data is available in the journal database.
The format will be familiar to those who are used to standard syslog logging. However, this actually collects data from more sources than traditional syslog implementations are capable of. It includes logs from the early boot process, the kernel, the initrd, and application standard error and out. These are all available in the journal.
You may notice that all of the timestamps being displayed are local time. This is available for every log entry now that we have our local time set correctly on our system. All of the logs are displayed using this new information.
If you want to display the timestamps in UTC, you can use the --utc flag:
journalctl --utc
While having access to such a large collection of data is definitely useful, such a large amount of information can be difficult or impossible to inspect and process mentally. Because of this, one of the most important features of journalctl is its filtering options.
The most basic of these which you might use daily, is the -b flag. This will show you all of the journal entries that have been collected since the most recent reboot.
journalctl -b
This will help you identify and manage information that is pertinent to your current environment.
In cases where you aren’t using this feature and are displaying more than one day of boots, you will see that journalctl has inserted a line that looks like this whenever the system went down:
. . .
-- Reboot --
. . .
This can be used to help you logically separate the information into boot sessions.
While you will commonly want to display the information from the current boot, there are certainly times when past boots would be helpful as well. The journal can save information from many previous boots, so journalctl can be made to display information easily.
Some distributions enable saving previous boot information by default, while others disable this feature. To enable persistent boot information, you can either create the directory to store the journal by typing:
sudo mkdir -p /var/log/journal
Or you can edit the journal configuration file:
sudo nano /etc/systemd/journald.conf
Under the [Journal] section, set the Storage= option to “persistent” to enable persistent logging:
/etc/systemd/journald.conf
. . .
[Journal]
Storage=persistent
When saving previous boots is enabled on your server, journalctl provides some commands to help you work with boots as a unit of division. To see the boots that journald knows about, use the --list-boots option with journalctl:
journalctl --list-boots
-2 caf0524a1d394ce0bdbcff75b94444fe Tue 2015-02-03 21:48:52 UTC—Tue 2015-02-03 22:17:00 UTC
-1 13883d180dc0420db0abcb5fa26d6198 Tue 2015-02-03 22:17:03 UTC—Tue 2015-02-03 22:19:08 UTC
0 bed718b17a73415fade0e4e7f4bea609 Tue 2015-02-03 22:19:12 UTC—Tue 2015-02-03 23:01:01 UTC
This will display a line for each boot. The first column is the offset for the boot that can be used to easily reference the boot with journalctl. If you need an absolute reference, the boot ID is in the second column. You can tell the time that the boot session refers to with the two time specifications listed towards the end.
To display information from these boots, you can use information from either the first or second column.
For instance, to see the journal from the previous boot, use the -1 relative pointer with the -b flag:
journalctl -b -1
You can also use the boot ID to call back the data from a boot:
journalctl -b caf0524a1d394ce0bdbcff75b94444fe
While seeing log entries by boot is incredibly useful, often you may wish to request windows of time that do not align well with system boots. This may be especially true when dealing with long-running servers with significant uptime.
You can filter by arbitrary time limits using the --since and --until options, which restrict the entries displayed to those after or before the given time, respectively.
The time values can come in a variety of formats. For absolute time values, you should use the following format:
YYYY-MM-DD HH:MM:SS
For instance, we can see all of the entries since January 10th, 2015 at 5:15 PM by typing:
journalctl --since "2015-01-10 17:15:00"
If components of the above format are left off, some defaults will be applied. For instance, if the date is omitted, the current date will be assumed. If the time component is missing, “00:00:00” (midnight) will be substituted. The seconds field can be left off as well to default to “00”:
journalctl --since "2015-01-10" --until "2015-01-11 03:00"
The journal also understands some relative values and named shortcuts. For instance, you can use the words “yesterday”, “today”, “tomorrow”, or “now”. You do relative times by prepending “-” or “+” to a numbered value or using words like “ago” in a sentence construction.
To get the data from yesterday, you could type:
journalctl --since yesterday
If you received reports of a service interruption starting at 9:00 AM and continuing until an hour ago, you could type:
journalctl --since 09:00 --until "1 hour ago"
As you can see, it’s relatively easy to define flexible windows of time to filter the entries you wish to see.
We learned above some ways that you can filter the journal data using time constraints. In this section we’ll discuss how to filter based on what service or component you are interested in. The systemd journal provides a variety of ways of doing this.
Perhaps the most useful way of filtering is by the unit you are interested in. We can use the -u option to filter in this way.
For instance, to see all of the logs from an Nginx unit on our system, we can type:
journalctl -u nginx.service
Typically, you would probably want to filter by time as well in order to display the lines you are interested in. For instance, to check on how the service is running today, you can type:
journalctl -u nginx.service --since today
This type of focus becomes extremely helpful when you take advantage of the journal’s ability to interleave records from various units. For instance, if your Nginx process is connected to a PHP-FPM unit to process dynamic content, you can merge the entries from both in chronological order by specifying both units:
journalctl -u nginx.service -u php-fpm.service --since today
This can make it much easier to spot the interactions between different programs and debug systems instead of individual processes.
Some services spawn a variety of child processes to do work. If you have scouted out the exact PID of the process you are interested in, you can filter by that as well.
To do this we can filter by specifying the _PID field. For instance if the PID we’re interested in is 8088, we could type:
journalctl _PID=8088
At other times, you may wish to show all of the entries logged from a specific user or group. This can be done with the _UID or _GID filters. For instance, if your web server runs under the www-data user, you can find the user ID by typing:
id -u www-data
33
Afterwards, you can use the ID that was returned to filter the journal results:
journalctl _UID=33 --since today
The systemd journal has many fields that can be used for filtering. Some of those are passed from the process being logged and some are applied by journald using information it gathers from the system at the time of the log.
The leading underscore indicates that the _PID field is of the latter type. The journal automatically records and indexes the PID of the process that is logging for later filtering. You can find out about all of the available journal fields by typing:
man systemd.journal-fields
We will be discussing some of these in this guide. For now though, we will go over one more useful option having to do with filtering by these fields. The -F option can be used to show all of the available values for a given journal field.
For instance, to see which group IDs the systemd journal has entries for, you can type:
journalctl -F _GID
32
99
102
133
81
84
100
0
124
87
This will show you all of the values that the journal has stored for the group ID field. This can help you construct your filters.
We can also filter by providing a path location.
If the path leads to an executable, journalctl will display all of the entries that involve the executable in question. For instance, to find those entries that involve the bash executable, you can type:
journalctl /usr/bin/bash
Usually, if a unit is available for the executable, that method is cleaner and provides better info (entries from associated child processes, etc). Sometimes, however, this is not possible.
Kernel messages, those usually found in dmesg output, can be retrieved from the journal as well.
To display only these messages, we can add the -k or --dmesg flags to our command:
journalctl -k
By default, this will display the kernel messages from the current boot. You can specify an alternative boot using the normal boot selection flags discussed previously. For instance, to get the messages from five boots ago, you could type:
journalctl -k -b -5
One filter that system administrators often are interested in is the message priority. While it is often useful to log information at a very verbose level, when actually digesting the available information, low priority logs can be distracting and confusing.
You can use journalctl to display only messages of a specified priority or above by using the -p option. This allows you to filter out lower priority messages.
For instance, to show only entries logged at the error level or above, you can type:
journalctl -p err -b
This will show you all messages marked as error, critical, alert, or emergency. The journal implements the standard syslog message levels. You can use either the priority name or its corresponding numeric value. In order of highest to lowest priority, these are:
•.0: emerg
•.1: alert
•.2: crit
•.3: err
•.4: warning
•.5: notice
•.6: info
•.7: debug
The above numbers or names can be used interchangeably with the -p option. Selecting a priority will display messages marked at the specified level and those above it.
Above, we demonstrated entry selection through filtering. There are other ways we can modify the output though. We can adjust the journalctl display to fit various needs.
We can adjust how journalctl displays data by telling it to shrink or expand the output.
By default, journalctl will show the entire entry in the pager, allowing the entries to trail off to the right of the screen. This info can be accessed by pressing the right arrow key.
If you’d rather have the output truncated, inserting an ellipsis where information has been removed, you can use the --no-full option:
journalctl --no-full
. . .
Feb 04 20:54:13 journalme sshd[937]: Failed password for root from 83.234.207.60...h2
Feb 04 20:54:13 journalme sshd[937]: Connection closed by 83.234.207.60 [preauth]
Feb 04 20:54:13 journalme sshd[937]: PAM 2 more authentication failures; logname...ot
You can also go in the opposite direction with this and tell journalctl to display all of its information, regardless of whether it includes unprintable characters. We can do this with the -a flag:
journalctl -a
By default, journalctl displays output in a pager for easier consumption. If you are planning on processing the data with text manipulation tools, however, you probably want to be able to output to standard output.
You can do this with the --no-pager option:
journalctl --no-pager
This can be piped immediately into a processing utility or redirected into a file on disk, depending on your needs.
If you are processing journal entries, as mentioned above, you most likely will have an easier time parsing the data if it is in a more consumable format. Luckily, the journal can be displayed in a variety of formats as needed. You can do this using the -o option with a format specifier.
For instance, you can output the journal entries in JSON by typing:
journalctl -b -u nginx -o json
{ "__CURSOR" : "s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635", "__REALTIME_TIMESTAMP" : "1422990364739502", "__MONOTONIC_TIMESTAMP" : "27200938", "_BOOT_ID" : "81b58db8fd9046ab9f847ddb82a2fa2d", "PRIORITY" : "6", "_UID" : "0", "_GID" : "0", "_CAP_EFFECTIVE" : "3fffffffff", "_MACHINE_ID" : "752737531a9d1a9c1e3cb52a4ab967ee", "_HOSTNAME" : "desktop", "SYSLOG_FACILITY" : "3", "CODE_FILE" : "src/core/unit.c", "CODE_LINE" : "1402", "CODE_FUNCTION" : "unit_status_log_starting_stopping_reloading", "SYSLOG_IDENTIFIER" : "systemd", "MESSAGE_ID" : "7d4958e842da4a758f6c1cdc7b36dcc5", "_TRANSPORT" : "journal", "_PID" : "1", "_COMM" : "systemd", "_EXE" : "/usr/lib/systemd/systemd", "_CMDLINE" : "/usr/lib/systemd/systemd", "_SYSTEMD_CGROUP" : "/", "UNIT" : "nginx.service", "MESSAGE" : "Starting A high performance web server and a reverse proxy server...", "_SOURCE_REALTIME_TIMESTAMP" : "1422990364737973" }
. . .
This is useful for parsing with utilities. You could use the json-pretty format to get a better handle on the data structure before passing it off to the JSON consumer:
journalctl -b -u nginx -o json-pretty
{
"__CURSOR" : "s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635",
"__REALTIME_TIMESTAMP" : "1422990364739502",
"__MONOTONIC_TIMESTAMP" : "27200938",
"_BOOT_ID" : "81b58db8fd9046ab9f847ddb82a2fa2d",
"PRIORITY" : "6",
"_UID" : "0",
"_GID" : "0",
"_CAP_EFFECTIVE" : "3fffffffff",
"_MACHINE_ID" : "752737531a9d1a9c1e3cb52a4ab967ee",
"_HOSTNAME" : "desktop",
"SYSLOG_FACILITY" : "3",
"CODE_FILE" : "src/core/unit.c",
"CODE_LINE" : "1402",
"CODE_FUNCTION" : "unit_status_log_starting_stopping_reloading",
"SYSLOG_IDENTIFIER" : "systemd",
"MESSAGE_ID" : "7d4958e842da4a758f6c1cdc7b36dcc5",
"_TRANSPORT" : "journal",
"_PID" : "1",
"_COMM" : "systemd",
"_EXE" : "/usr/lib/systemd/systemd",
"_CMDLINE" : "/usr/lib/systemd/systemd",
"_SYSTEMD_CGROUP" : "/",
"UNIT" : "nginx.service",
"MESSAGE" : "Starting A high performance web server and a reverse proxy server...",
"_SOURCE_REALTIME_TIMESTAMP" : "1422990364737973"
}
. . .
The following formats can be used for display:
•.cat: Displays only the message field itself.
•.export: A binary format suitable for transferring or backing up.
•.json: Standard JSON with one entry per line.
•.json-pretty: JSON formatted for better human-readability
•.json-sse: JSON formatted output wrapped to make add server-sent event compatible
•.short: The default syslog style output
•.short-iso: The default format augmented to show ISO 8601 wallclock timestamps.
•.short-monotonic: The default format with monotonic timestamps.
•.short-precise: The default format with microsecond precision
•.verbose: Shows every journal field available for the entry, including those usually hidden internally.
These options allow you to display the journal entries in the whatever format best suits your current needs.
The journalctl command imitates how many administrators use tail for monitoring active or recent activity. This functionality is built into journalctl, allowing you to access these features without having to pipe to another tool.
To display a set amount of records, you can use the -n option, which works exactly as tail -n.
By default, it will display the most recent 10 entries:
journalctl -n
You can specify the number of entries you’d like to see with a number after the -n:
journalctl -n 20
To actively follow the logs as they are being written, you can use the -f flag. Again, this works as you might expect if you have experience using tail -f:
journalctl -f
You may be wondering about the cost is of storing all of the data we’ve seen so far. Furthermore, you may be interesting in cleaning up some older logs and freeing up space.
You can find out the amount of space that the journal is currently occupying on disk by using the --disk-usage flag:
journalctl --disk-usage
Journals take up 8.0M on disk.
If you wish to shrink your journal, you can do that in two different ways (available with systemd version 218 and later).
If you use the --vacuum-size option, you can shrink your journal by indicating a size. This will remove old entries until the total journal space taken up on disk is at the requested size:
sudo journalctl --vacuum-size=1G
Another way that you can shrink the journal is providing a cutoff time with the --vacuum-time option. Any entries beyond that time are deleted. This allows you to keep the entries that have been created after a specific time.
For instance, to keep entries from the last year, you can type:
sudo journalctl --vacuum-time=1years
You can configure your server to place limits on how much space the journal can take up. This can be done by editing the /etc/systemd/journald.conf file.
The following items can be used to limit the journal growth:
•.SystemMaxUse=: Specifies the maximum disk space that can be used by the journal in persistent storage.
•.SystemKeepFree=: Specifies the amount of space that the journal should leave free when adding journal entries to persistent storage.
•.SystemMaxFileSize=: Controls how large individual journal files can grow to in persistent storage before being rotated.
•.RuntimeMaxUse=: Specifies the maximum disk space that can be used in volatile storage (within the /run filesystem).
•.RuntimeKeepFree=: Specifies the amount of space to be set aside for other uses when writing data to volatile storage (within the /run filesystem).
•.RuntimeMaxFileSize=: Specifies the amount of space that an individual journal file can take up in volatile storage (within the /run filesystem) before being rotated.
By setting these values, you can control how journald consumes and preserves space on your server. Keep in mind that SystemMaxFileSize and RuntimeMaxFileSize will target archived files to reach stated limits. This is important to remember when interpreting file counts after a vacuuming operation.
As you can see, the systemd journal is incredibly useful for collecting and managing your system and application data. Most of the flexibility comes from the extensive metadata automatically recorded and the centralized nature of the log. The journalctl command makes it easy to take advantage of the advanced features of the journal and to do extensive analysis and relational debugging of different application components.
systemd possède son propre mécanisme de journalisation, syslog n'est plus requis par défaut.
Le paquet rsyslog est toujours installable, mais pourquoi utiliser 2 mêmes outils sur un même système ?
Voici quelques commandes utiles pour se repérer ....
Seul le root et le groupe systemd-journal peuvent visualiser le journal.
Pour voir les logs :
journalctl
mai 21 21:24:21 localhost.localdomain systemd[2396]: Startup finished in 1.216s.
mai 21 21:24:21 localhost.localdomain sshd[1695]: pam_tcb(sshd:session): Session opened fo
mai 21 21:24:24 localhost.localdomain su[2452]: pam_tcb(su:auth): Authentication passed fo
mai 21 21:24:24 localhost.localdomain su[2452]: (to root) adrien on pts/0
mai 21 21:24:24 localhost.localdomain su[2452]: pam_tcb(su:session): Session opened for ro
De manière "tail -f", continue sur la console jusqu'à ctrl+C :
journalctl -f
-- Logs begin at dim. 2015-03-29 22:59:55 CEST. --
mai 21 21:24:21 localhost.localdomain systemd[2396]: Starting Default.
mai 21 21:24:21 localhost.localdomain systemd[2396]: Reached target Default.
mai 21 21:24:21 localhost.localdomain systemd[2396]: Startup finished in 1.216s.
mai 21 21:24:21 localhost.localdomain sshd[1695]: pam_tcb(sshd:session): Session opened fo
Filtrer par service
journalctl -u crond
mai 21 21:23:43 localhost.localdomain crond[624]: (CRON) INFO (Syslog will be used instead
mai 21 21:23:43 localhost.localdomain crond[624]: (CRON) INFO (RANDOM_DELAY will be scaled
mai 21 21:23:44 localhost.localdomain crond[624]: (CRON) INFO (running with inotify suppor
Filtrer par PID :
journalctl _PID=1
mai 21 21:23:51 localhost.localdomain systemd[1]: lm_sensors.service failed.
mai 21 21:24:03 localhost.localdomain systemd[1]: plymouth-quit-wait.service start operati
mai 21 21:24:03 localhost.localdomain systemd[1]: Failed to start Wait for Plymouth Boot S
mai 21 21:24:03 localhost.localdomain systemd[1]: Unit plymouth-quit-wait.service entered
mai 21 21:24:03 localhost.localdomain systemd[1]: plymouth-quit-wait.service failed.
Filtrer par programme :
journalctl /usr/sbin/sshd
mai 21 21:24:06 localhost.localdomain sshd[1268]: Server listening on 0.0.0.0 port 22.
mai 21 21:24:06 localhost.localdomain sshd[1268]: Server listening on :: port 22.
mai 21 21:24:19 localhost.localdomain sshd[1836]: pam_tcb(sshd:auth): Authentication passe
mai 21 21:24:19 localhost.localdomain sshd[1695]: Accepted keyboard-interactive/pam for ad
mai 21 21:24:21 localhost.localdomain sshd[1695]: pam_tcb(sshd:session): Session opened fo
Filtrer par niveau de log (ici que les erreurs) :
journalctl -p err
mai 21 21:24:06 localhost.localdomain rpcbind[1278]: Cannot open '/var/lib/rpcbind/rpcbind
mai 21 21:24:06 localhost.localdomain rpcbind[1278]: Cannot open '/var/lib/rpcbind/portmap
mai 21 21:24:14 localhost.localdomain kernel: xt_addrtype: ipv6 does not support BROADCAS
On peut évidemment combiner toutes ces options :
journalctl -f /usr/sbin/urpmi -p info
-- Logs begin at dim. 2015-03-29 22:59:55 CEST. --
mai 21 21:19:29 localhost.localdomain [RPM][9040]: erase lib64kwineffects1-2:4.11.16-2.mga5.x86_64: success
mai 21 21:19:29 localhost.localdomain [RPM][9040]: erase lib64kwinglesutils1-2:4.11.16-2.mga5.x86_64: success
mai 21 21:19:29 localhost.localdomain [RPM][9040]: Transaction ID 555e2fb9 finished: 0
- - Reboot - -
mai 21 21:34:03 localhost.localdomain urpmi[8355]: called with: hnghggjh
On peut aussi filtrer les logs par date avec --since et --until. Il faut mettre la date au format YYYY-MM-DD HH:MM:SS
Par exemple depuis 21h le 10/02/2016 :
journalctl --since "2016-02-10 21:00:00"
Ou jusqu'à 22h le 10/02/2016 :
journalctl --until "2016-02-10 22:00:00"
Ou dans cet intervalle en combinant les deux :
journalctl --since "2016-02-10 21:00:00" --until "2016-02-10 22:00:00"
On peut parfois utiliser les dates avec des adverbes de temps (exemple depuis hier)
journalctl --since yesterday
Ou une durée (il y a 10 minutes) :
journalctl --since "10 minutes ago"
On peut imiter la taille maximum du journal (par défaut à 10% de la taille du système de fichier). Pour la fixer à 500 Mo par exemple, on édite le fichier /etc/systemd/journald.conf :
SystemMaxUse=500M
En fixant une limite par fichier, on a un équivalent de logrotate (par défaut, il garde 7 rotations) :
SystemMaxUse=500M
SystemMaxFileSize=100M
On peut aussi faire en sorte que les journaux ne soient pas conservés sur le disque au redémarrage (machine non critique) :
[Journal]
Storage=volatile
Si on souhaite envoyer quand même les logs dans /var/log/messages, c'est possible !
Installer rsyslog et démarrer le service.
Ensuite, éditer le fichier /etc/systemd/journald.conf et modifier cette option :
ForwardToSyslog=yes